fix(podman): resolve macOS Podman socket dynamically by gmenher · Pull Request #3135 · NVIDIA/OpenShell

gmenher · 2026-09-02T16:33:02Z

Summary

This PR removes the last macOS Podman socket assumption from the driver-config volume e2e path and makes the Podman-backed e2e harness respect the socket reported by podman machine inspect.

On newer macOS Podman setups, the legacy ~/.local/share/containers/podman/machine/podman.sock path is not a reliable source of truth after restarting podman machine. That meant the driver-config volume helper and the Podman e2e flow could fail even when the VM was healthy and reachable through the socket Podman itself reports.

Related Issue

Closes #1690

Changes

Prefer OPENSHELL_PODMAN_SOCKET when it is already set, so explicit user overrides still win.
On macOS, resolve the Podman API socket from podman machine inspect before falling back to the legacy socket path.
Export CONTAINER_HOST whenever the Podman socket is resolved so child processes and follow-on commands use the same endpoint.
Make the Podman e2e preflight use the resolved socket consistently instead of assuming the legacy socket layout.
Add focused tests for parsing macOS podman machine inspect output.
Tighten a couple of Bash array expansions in the e2e and prebuilt-binary scripts so empty-array cases stay safe.

While validating the macOS Podman Machine change, I hit two unrelated harness bugs in gateway-common.sh and stage-prebuilt-binaries.sh caused by set -u plus empty arrays. I included those fixes because they blocked local e2e verification, but they are separate from the macOS socket change.

Testing

mise run pre-commit passes
Unit tests added/updated
E2E coverage updated and validated for the Podman lane
Manual smoke test after restarting podman machine confirmed the full user flow

Checklist

Follows Conventional Commits

copy-pr-bot · 2026-09-02T16:33:05Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-09-02T16:33:15Z

All contributors have signed the DCO ✍️ ✅
_{Posted by the DCO Assistant Lite bot.}

gmenher · 2026-09-02T16:39:42Z

I have read the DCO document and I hereby sign the DCO.

johntmyers

gator-agent

PR Review Status

The macOS Podman fix is project-valid and the focused socket changes are ready to continue, but the latest commit also removes a repository-advertised troubleshooting skill while leaving its callers and documentation intact.

Action required: restore .agents/skills/debug-openshell-cluster/SKILL.md, or move an intentional coordinated removal into a complete, separately scoped change.

Blocking findings:

GATOR-b92131c8-01: the deleted skill remains referenced by repository guidance and companion workflows.

Carried findings:

None

Gator metadata

Validation: Concentrated macOS Podman compatibility fix linked to triaged issue #1690
Docs: Fern compute-driver reference updated; existing reference-folder navigation covers the page
Checks: Current-head branch and Helm gates are waiting for the copy-PR mirror
E2E: Required for Podman runtime behavior; dispatch deferred until blocking review feedback is resolved
Head SHA: b92131c8a572bf935fd60a0f2784116b823ec676
Base SHA: 8bc79552638a625971f8c6d30c26efa917a19cf9
Merge base SHA: 06f0aa6fe30177f2aecba9af396649adae4d7cdd
Patch ID: 4bae9c48f63693d1ea65d2a789486c786b681b6d
Gator payload: 8
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:in-review

github-actions · 2026-09-02T17:21:21Z

Label test:e2e applied, but pull-request/3135 does not exist yet. A maintainer needs to comment /ok to test 69bf396d84f2b52f2b96b5c4e24e237211913560 to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

johntmyers · 2026-09-02T17:21:39Z

/ok to test 69bf396

johntmyers

gator-agent

PR Review Status

The author-only follow-up resolves GATOR-b92131c8-01: the effective PR tree keeps the canonical public troubleshooting skill and applies the intended macOS Podman guidance there. The bounded delta review found no new blocking findings, and the required Branch Checks, Helm Lint, and E2E workflows are queued for the current head.

Blocking findings:

No blocking findings remain

Carried findings:

None; GATOR-b92131c8-01 is resolved

Gator metadata

Validation: Concentrated macOS Podman compatibility fix linked to triaged issue #1690
Docs: Fern compute-driver reference and canonical public troubleshooting skill are updated; existing reference navigation covers the page
Checks: Current-head Branch Checks and Helm Lint workflows are queued
E2E: test:e2e applied; /ok to test 69bf396d84f2b52f2b96b5c4e24e237211913560 posted; current-head Branch E2E workflow is queued
Head SHA: 69bf396d84f2b52f2b96b5c4e24e237211913560
Base SHA: 8bc79552638a625971f8c6d30c26efa917a19cf9
Merge base SHA: 06f0aa6fe30177f2aecba9af396649adae4d7cdd
Patch ID: a80112376cbab89bacc549dc61b77904c9dfd8ba
Gator payload: 8
Review mode: follow_up
Previous reviewed SHA: b92131c8a572bf935fd60a0f2784116b823ec676
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:watch-pipeline

elezar

Can we remove the unrelated skills changes from this PR and keep it focusses on the macos-specific podman machine changes.

johntmyers

gator-agent

PR Review Status

Thanks @elezar. I checked your request to keep this focused on the macOS Podman Machine change against the two author commits and the effective patch after the merge from main. The unrelated E2E shell edits net out against the previously reviewed head, and the bounded delta review found no new blocking code defects, but skills/debug-openshell-cluster/SKILL.md still has the nine-line addition you asked to remove.

Action required: @gmenher, remove the remaining skills/debug-openshell-cluster/SKILL.md change, or obtain an explicit maintainer decision to retain it in this PR.

Blocking findings:

No new Gator blocking findings

Carried feedback:

@elezar's requested scope change remains unresolved

Gator metadata

Validation: Concentrated macOS Podman compatibility fix linked to validated issue #1690
Docs: Fern compute-driver reference is updated; maintainer-requested skill scope cleanup remains
Checks: Current-head required checks are pending; pipeline handoff is deferred until review feedback is resolved
E2E: test:e2e remains required; current-head dispatch is deferred until review feedback is resolved
Head SHA: 39f5ede5df3633a5e785254457dc61d38591855a
Base SHA: 0f0c94bda3a2056bd3413accd13e361488288cc4
Merge base SHA: 0f0c94bda3a2056bd3413accd13e361488288cc4
Patch ID: a7eba7991fb1b953ad1fd0497a1bceef33f91f79
Gator payload: 8
Review mode: follow_up
Previous reviewed SHA: 69bf396d84f2b52f2b96b5c4e24e237211913560
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:in-review

elezar

Thanks for updating this. I think this needs to be rebased off main before we can continue reviewing it, though.\n\nMy one concern is that we're adding system-specific behavior into the test code instead of reusing the socket that was already resolved in the helper scripts.

On macOS, Homebrew-installed Podman does not create the default socket path that the Podman driver probes. Document the OPENSHELL_PODMAN_SOCKET override and the podman machine inspect lookup in both the compute drivers reference and the debug-openshell-cluster skill. Fixes NVIDIA#1690 Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

johntmyers · 2026-09-04T11:09:37Z

/ok to test f841e74

github-actions · 2026-09-04T11:10:13Z

Label test:e2e applied for f841e74. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

johntmyers

gator-agent

PR Review Status

Thanks @elezar. I checked your request to rebase and reuse the socket resolved by the helper scripts against current head f841e7428690121b3c989b4787bd39778b631de6: the branch now sits on current main, the unrelated skill edit is gone, and driver_config_volume.rs now only consumes OPENSHELL_PODMAN_SOCKET after with-podman-gateway.sh resolves and exports it. The critical-only delta review found no new Critical defects, and the required current-head workflows are running.

Blocking findings:

No blocking findings remain

Carried findings:

None; GATOR-b92131c8-01 remains resolved, and @elezar's latest scope and socket-reuse feedback is addressed by the current patch

Gator metadata

Validation: Concentrated macOS Podman compatibility fix linked to issue #1690
Docs: Fern compute-driver reference updated; existing reference navigation covers the page
Checks: Current-head Branch Checks are queued/running and Helm Lint is green
E2E: test:e2e re-applied after /ok to test f841e7428690121b3c989b4787bd39778b631de6; current-head Branch E2E is queued/running
Head SHA: f841e7428690121b3c989b4787bd39778b631de6
Base SHA: c93b2fa7da3a40a6dd6c077973cb13d2755f7e50
Merge base SHA: c93b2fa7da3a40a6dd6c077973cb13d2755f7e50
Patch ID: 0475e241149067e0aa5fffd1421cae632c61155d
Gator payload: 8
Review mode: critical_only
Previous reviewed SHA: 39f5ede5df3633a5e785254457dc61d38591855a
Review budget exhausted: yes
Maintainer decision required: no
Next state: gator:watch-pipeline

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

johntmyers · 2026-09-04T14:40:15Z

/ok to test 31fc1a4

johntmyers · 2026-09-04T15:15:06Z

gator-agent

Maintainer Approval Needed

Gator validation and PR monitoring are complete. The current head is patch-equivalent to the previously reviewed head, the prior Gator finding remains resolved, and the maintainer's macOS Podman scope and socket-reuse feedback is addressed.

Review: No blocking Gator findings remain. The existing changes-requested review now needs a maintainer to revisit it and approve if satisfied.

Human maintainer approval is now required.

Gator metadata

Validation: Concentrated macOS Podman compatibility fix linked to issue docs: document macOS Podman socket path mismatch and dynamic lookup #1690
Docs: Fern compute-driver reference updated; existing reference navigation covers the page
Checks: OpenShell / Branch Checks and OpenShell / Helm Lint are green for the current head
E2E: test:e2e is applied and OpenShell / E2E is green for the current head
Head SHA: 31fc1a46a24406ac10f68b6b8a2352161abd84b4
Patch ID: 0475e241149067e0aa5fffd1421cae632c61155d
Gator payload: 8
Review mode: already_reviewed (rebase-equivalent effective patch)
Review budget exhausted: yes
Maintainer decision required: no
Next state: gator:approval-needed

gmenher requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners September 2, 2026 16:33

johntmyers reviewed Sep 2, 2026

View reviewed changes

Comment thread .agents/skills/debug-openshell-cluster/SKILL.md

johntmyers added gator:in-review Gator is reviewing or awaiting PR review feedback test:e2e Requires end-to-end coverage labels Sep 2, 2026

johntmyers reviewed Sep 2, 2026

View reviewed changes

johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:in-review Gator is reviewing or awaiting PR review feedback gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 2, 2026

elezar requested changes Sep 3, 2026

View reviewed changes

gmenher requested a review from elezar September 3, 2026 16:47

johntmyers reviewed Sep 3, 2026

View reviewed changes

johntmyers added gator:in-review Gator is reviewing or awaiting PR review feedback and removed gator:approval-needed Gator completed review; maintainer approval needed labels Sep 3, 2026

elezar requested changes Sep 4, 2026

View reviewed changes

gmenher added 7 commits September 4, 2026 11:17

fix(podman): resolve macOS Podman socket dynamically

685eff2

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

chore: restore debug skill file

a5b37bf

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

chore: drop legacy debug skill path

63d8f7f

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

fix(podman): trim unrelated e2e changes

e47bcf9

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

fix(e2e): harden shell array expansion

e1878b1

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

chore: remove unrelated skill note

f841e74

gmenher force-pushed the docs/macos-podman-socket-1690 branch from 39f5ede to f841e74 Compare September 4, 2026 10:51

johntmyers added test:e2e Requires end-to-end coverage and removed test:e2e Requires end-to-end coverage labels Sep 4, 2026

johntmyers reviewed Sep 4, 2026

View reviewed changes

johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:blocked Gator is blocked by process or repository gates and removed gator:in-review Gator is reviewing or awaiting PR review feedback gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 4, 2026

ci: retrigger checks

31fc1a4

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:blocked Gator is blocked by process or repository gates gator:watch-pipeline Gator is monitoring PR CI/CD status labels Sep 4, 2026

Uh oh!

Conversation

gmenher commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Sep 2, 2026

Uh oh!

github-actions Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gmenher commented Sep 2, 2026

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

johntmyers commented Sep 2, 2026

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

elezar left a comment

Choose a reason for hiding this comment

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

elezar left a comment

Choose a reason for hiding this comment

Uh oh!

johntmyers commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

johntmyers left a comment

Choose a reason for hiding this comment

PR Review Status

Uh oh!

johntmyers commented Sep 4, 2026

Uh oh!

johntmyers commented Sep 4, 2026

Maintainer Approval Needed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gmenher commented Sep 2, 2026 •

edited

Loading

github-actions Bot commented Sep 2, 2026 •

edited

Loading